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Abstract 


This  study  aims  to  elucidate  the  figurative  relations  between  spatial  significants,  such 
as  geographical  objects,  and  the  visual-graphical  (hand)  signifiers  that  commanders  use  to 
designate  or  generate  these  significants  on  a  map  during  a  tactical  planning  task. 
Specifically,  the  objective  is  to  specify  whether  visual-graphical  signifiers  have  a  physical 
property,  such  as  a  geometrical  form,  that  reproduces  a  corresponding  property  of  their 
significants.  This  study  is  part  of  a  broader  project  aimed  at  specifying  the  symbolic 
(lexical,  semantic,  and  figurative)  properties  of  graphical  actions  in  multimodal  dialogues 
over  a  map  during  tactical  planning.  Knowledge  of  these  properties  will  be  used  in  the 
design  of  multimodal  interfaces  aimed  at  recognising  and  interpreting  such  graphical 
actions.  The  design  of  multimodal  interfaces  is  in  turn  part  of  the  development  of  digitized 
land  forces. 

Seven  commanding  officers  provided  verbal  and  graphical  estimates  of  a  tactical 
scenario  portrayed  on  a  topographic  map.  These  sessions  were  videotaped.  The  graphical 
component  consisted  of  graphical  (hand)  actions  which  referred  to  significants  on  a 
topographic  map.  We  identified  these  graphical  actions  as  visual-graphical  signifiers  in  a 
lexicon.  We  also  determined  a  set  of  significants  (objects,  concepts,  attributes,  and 
relations)  from  the  analysis  of  the  commanders'  dialogue  in  their  verbal  and  graphical 
components.  The  data  for  this  study  were  the  lexicon  of  visual-graphical  signifiers  and  the 
significants  related  to  these  signifiers.  To  elucidate  the  figurative  properties  of  the  visual- 
graphical  signifiers,  we  classified  the  significants  according  to  their  geometrical  properties 
(one  dimension,  two  dimensions,  and  nonspecified  dimensionality),  and  then  grouped 
them  according  to  the  type  of  visual-graphical  signifier  to  which  they  were  related.  The 
results  indicate  that  visual-graphical  signifiers  figuratively  relate  to  their  significants’ 
geometrical  properties.  The  figurative  properties  of  the  visual-graphical  signifiers  suggest 
that  commanders  use  these  signifiers  to  depict  significants  on  a  map,  and  to  represent  the 
visual  properties  of  those  not  easily  nor  completly  conveyed  through  speech. 


-l- 


Executive  Summary 


Computers  are  now  being  used  to  support  tactical  planning  and  decision  making 
through  intelligent  dialogue  with  humans.  Military  planners  communicate  with  each  other 
using  both  speech  and  graphical  (hand)  actions  especially  while  discussing  problems 
involving  spatial  data,  such  as  geographical  objects  or  concepts.  The  multimodal  properties 
of  their  dialogue  should  also  be  possible  at  the  interface  between  a  human  and  an  intelligent 
computer  agent.  This  is  the  approach  that  several  military  applications  have  adopted  for  the 
design  of  multimodal  interfaces  capable  of  recognising  and  interpreting  speech  and 
graphical  actions  concurrently.  Multimodal  interfaces  constitute  an  intrinsic  component  of 
digitized  command  and  control  forces. 

However,  multimodal  interfaces  have  been  typically  limited  to  recognising  and 
interpreting  pointing  actions,  only  one  of  the  many  different  types  of  graphical  actions  that 
humans  use  to  denote  significants,  such  as  objects  or  concepts,  on  graphical  media.  This 
limitation  is  partly  related  to  the  lack  of  information  concerning  the  symbolic  properties 
(lexical,  semantic,  figurative)  of  graphical  actions  in  human  dialogue,  and  the  extent  to 
which  different  individuals  consistently  generate  these  properties.  Both  sources  of 
information  are  required  for  successful  design  of  graphical  interfaces  which  are  a  structural 
part  of  multimodal  interfaces.  As  Rhyne  and  Wolf  (1986)  argue,  the  real  productivity 
benefits  of  graphical  interfaces  depend  on  users'  consistent  use  of  graphical  actions  as  this 
attribute  increases  the  likelihood  of  their  reliable  recognition.  There  are  two  kinds  of 
consistency  that  affects  the  efficiency  of  graphical  interfaces:  inter-subject  consistency,  the 
extent  to  which  there  is  commonality  across  individuals  in  the  symbolic  properties  of 
graphical  actions,  such  as  their  geometrical  form  and  meaning;  and  intra-subject 
consistency,  the  extent  to  which  each  individual  reproduces  a  given  action  in  the  same 
geometrical  form  and  for  the  same  meaning. 

Based  on  a  tactical  planning  task  over  a  map,  we  have  shown  in  two  previous  studies 
that  graphical  actions  have  lexical  and  semantic  properties,  and  thus  constitute  signifiers. 
This  study  aims  to  specify  the  figurative  properties  of  the  signifiers  that  commanders 
produced  to  designate  or  generate  significants  on  a  map  during  the  tactical  planning  task. 
The  figurative  properties  would  imply  that  graphical  actions  have  a  geometrical  form  that 
reproduces  a  corresponding  geometrical  form  of  their  significant. 

Seven  commanding  officers  provided  verbal  and  graphical  estimates  of  a  tactical 
scenario  portrayed  on  a  topographic  map.  These  sessions  were  videotaped.  The  graphical 


component  consisted  of  graphical  (hand)  actions  involving  a  reference  to  significants  on  the 
topographic  map.  From  the  analysis  of  the  commanders  dialogues,  we  identified  a  lexicon 
of  graphical  actions,  and  the  set  of  significants  (objects,  concepts,  attributes,  and  relations) 
related  to  these  actions.  We  used  both  data  sets  for  this  study. 

To  identify  the  figurative  properties  of  the  graphical  actions,  we  classified  the 
significants  according  to  their  geometrical  attributes  (one  dimension,  two  dimensions,  and 
nonspecified  dimensionality),  and  then  grouped  them  according  to  the  type  of  graphical 
action  to  which  they  were  related.  The  results  indicate  that  graphical  actions  figuratively 
relate  to  their  significants'  geometrical  attributes.  The  figurative  properties  of  the  actions 
suggest  that  planners  use  such  actions  to  depict  significants  on  a  map,  and  to  represent  the 
visual  properties  of  those  not  easily  or  completly  conveyed  through  speech. 

The  results  support  the  utility  of  designing  graphical  interfaces  to  recognise  graphical 
actions.  The  graphical  interfaces  should  allow  the  user  to  employ  any  part  of  his  hand(s)  to 
denote  spatial  significants  on  a  two-dimensional  surface  as  observed  in  this  study,  but  also 
to  generate  significants  in  a  three-dimensional  space.  The  design  of  a  graphical  interface 
constitutes  an  intrinsic  component  of  a  multimodal  interface  capable  of  recognising  and 
interpreting  both  speech  and  graphical  actions  concurrently. 


1.  Introduction 


One  may  contend,  as  the  propositionalist  view  would  hold,  that  communication 
takes  place  essentially  through  natural  language  as  a  system  of  conventional  and  arbitrary 
signs  (Pylyshyn,  1973;  Palmer,  1975).  Multimodal  views  of  the  symbolic  representation 
(Kendon,  1985;  McNeill,  1987b;  McNeill  &  Levy,  1982;  Pavio,  1986;  Piaget  &  Inhelder, 
1963)  argue,  however,  that  communication  takes  place  through  the  exchange  of  signifiers 
which  include  not  only  signs  but  also  symbols.  Signifiers  are  symbolic  tokens  by  which 
humans  refer  to  significants1,  such  as  objects  or  concepts,  in  their  presence  or  represent 
them  in  their  absence  (Piaget,  1983;  Piaget  &  Inhelder,  1963;  Saussure,  1959).  For 
example,  one  can  refer  to  an  object  by  means  of  a  word  or  an  image.  In  their  strict  sense, 
signs  are  arbitrary  because  they  do  not  have  any  similarities  with  their  significant(s). 
Signs  include  verbal  signs  (oral  and  written  words),  scientific  signs,  and  braille  signs.  In 
contrast,  symbols  have  physical  properties  (acoustic,  visual,  and  haptic)  that  reproduce 
those  of  their  significant(s),  and  thus  have  a  figurative  function.  Symbols  include, 
deferred  forms  of  imitation,  graphical  representations  (e.g.,  map  symbols),  and  mental 
imagery  (Kosslyn  et  al.  1979;  Pavio,  1986;  Piaget,  1983;  Piaget  &  Inhelder,  1963). 
Humans  convey  signifiers  in  three  general  modes  depending  on  the  modality  or  sense- 
organ  used  (Boudreau  &  McCann,  1996):  acoustic  (vocal,  non  vocal),  visual  (graphical, 
non  graphical),  and  haptic  (graphical). 

Cognitive  theorists  (Ellis  &  Hunt,  1989;  Kendon,  1980, 1985;  McNeill,  1985, 
1987a,  b;  McNeill  &  Levy,  1982)  recognise  a  type  of  signifier  that  arises  from  the  visual 
modality,  i.e.,  from  hand  motions  distinct  from  those  involved  in  writing.  These  visual 
signifiers  are  capable  of  designating  significants  in  their  presence,  or  representing  them 
even  when  they  are  not  actually  perceived  at  the  time  they  are  signified.  For  example, 
descriptors  from  sign  languages  (Cuxac,  1991)  have  such  a  symbolic  function. 

Visual  signifiers  would  either  have  arbitrary  properties  in  relation  to  their 
significants,  as  in  the  case  of  formless  pointing  motions,  or  figurative  properties  in 
common  with  their  significants  as  in  the  case  of  drawing  motions.  Although  cognitive 
theorists  (see  Ellis  &  Hunt,  1989;  Kendon,  1972, 1980,  1985;  McNeill,  1985,  1987b) 
have  proposed  arguments  in  favor  of  the  figurative  properties  of  visual  signifiers,  few 
studies  (see  McNeill  &  Levy,  1982)  have  been  conducted  to  support  these  arguments. 


1  A  significant  refers  to  the  meaning  conveyed  by  a  signifier.  A  significant  may  consist  in  an  object, 
a  concept,  a  property  of  an  object  (or  concept),  or  a  relation  between  objects  (or  concepts). 


The  purpose  of  this  study  is  to  elucidate  the  figurative  properties  of  the  visual- 
graphical  signifiers  that  tactical  planners  use  to  designate  or  generate  significants  on  a 
map  during  a  tactical  planning  task.  By  virtue  of  these  properties,  the  visual  signifiers 
would  have  a  physical  attribute,  such  as  a  geometrical  form,  that  reproduces  a 
corresponding  attribute  of  their  significants). 

This  study  will  be  based  on  a  lexicon  of  visual-graphical  signifiers  developed  from 
the  tactical  planning  task  (Boudreau  &  McCann,  1994),  and  the  set  of  significants  related 
to  these  signifiers  (Boudreau  &  McCann,  1996).  In  the  following  section,  we  will 
describe  the  figurative  properties  of  signifiers  to  provide  a  theoretical  basis  upon  which  to 
elucidate  those  of  the  visual-graphical  signifiers. 

1.1  Figurative  Properties  of  Signifiers 

Signifiers  that  provide  an  approximate  reproduction  of  the  physical  attributes  of 
significants  are  said  to  have  figurative  properties  (Pavio,  1986;  Piaget,  1983).  Such 
signifiers  include:  imitation,  deferred  imitation,  mental  imagery,  and  descriptors  in  sign 
languages  (Cuxac,  1991;  Pavio,  1986).  Verbal  signifiers  (oral  and  written  words)  also 
have  figurative  properties  although  they  range  on  a  continuum  from  being  purely 
figurative  (e.g.,  onomatopoeia)  to  purely  arbitrary  (e.g.,  mathematical  signs).  In  contrast, 
mental  images2,  deferred  imitation,  and  imitation  are  essentially  figurative  (Jeannerod, 
1994;  Paivio,  1986;  Piaget,  1983). 

Mental  images  reproduce  significants  or  their  properties  when  they  are  no  longer 
perceived.  Psycho-physiological  studies  (e.g.,  Jeannerod,  1994;  Paivio,  1986;  Piaget  & 
Inhelder,  1963)  suggest  that  mental  images  constitute  internal  forms  of  imitation 
generated  from  previous  sensory-motor  activity.  For  example,  the  mental  image  of  a 
postural  movement  involves  a  motor  image  sketching  the  effective  production  of  the 
movement  (Jeannerod,  1994;  Piaget  &  Inhelder,  1963).  Reciprocally,  mental  images  also 
influence  imitation. 

Deferred  imitation  is  a  motor  action  (or  a  set  of  such  actions)  that  humans  use  to 
reproduce,  through  the  aid  of  memory,  a  significant  that  is  no  longer  physically  present. 
The  production  of  graphical  symbols  (Lee,  1991)  constitutes  a  particular  sort  of  deferred 


2  Arguments  have  been  made  against  the  hypothesis  that  mental  imagery  constitutes  a  pictorial 
representation  of  significants  (for  a  review  see  Anderson,  1978).  However,  Paivio's  (1986)  Dual-Code 
theory  of  mental  representation  has  reinstated  this  hypothesis,  and  Jeannerod  (1994)  has  shown  that  it  is 
compatible  with  recent  neuropsychological  models  of  such  mental  representation. 


imitation  in  that  the  hand  motion  results  in  some  visible  mark,  such  as  a  shape,  being  left 
on  a  graphical  medium.  Imitation  in  its  broad  sense  constitutes  a  motor  reproduction 
performed  in  the  presence  of  its  significant  rather  than  in  its  absence.  For  example,  a 
linear  motion  might  reproduce  the  linear  shape  of  a  route  symbolised  on  a  map. 

Deferred  imitation  and  imitation  include  a  type  of  signifier  called  visual  signifiers 
(Boudreau  &  McCann,  1993, 1996).  Humans  express  visual  signifiers  either  graphically 
or  nongraphically.  Visual- graphical  signifiers  are  those  that  involve  a  manual  (hand) 
reference  to  a  graphical  medium.  The  manual  reference  can  be  made  without  any  visible 
mark  being  left  on  the  medium.  An  example  is  the  tracing  of  a  linear  feature  on  a  map 
with  the  forefinger.  The  manual  reference  may,  on  the  other  hand,  result  in  some  visible 
mark  being  left  on  the  medium,  for  example,  a  shape  or  a  character  (letters,  numbers), 
being  the  result  of  drawing  or  writing.  Signifiers  that  have  a  visual  component  with  no 
manual  reference  to  a  graphical  medium  are  termed  visual-nongraphical  signifiers.  These 
include  iconographic  motions  (Efron,  1941),  physiographic  motions  (Freedman,  1977) 
spatial  motions  (Ekman  &  Friesen,  1972),  iconic  motions  (see  McNeill,  1987b;  McNeill 
&  Levy,  1982),  and  descriptors  from  sign  languages  (Cuxac,  1991).  According  to 
Freedman  (1977)  and  McNeill  (1987a,  b),  the  above  visual  signifiers  would  depict, 
through  overt  manual  activity,  the  mental  imagery  that  occurs  during  speech,  but  that  is 
not  easily  nor  completely  conveyed  through  speech.  For  example,  a  linear  motion  would 
represent  the  mental  image  of  an  avenue  of  approach.  Visual  signifiers  would  then 
symbolize  the  figurative  properties  of  these  mental  images3  as  well  as  imitate  those  of 
significants  that  are  physically  present. 

Humans  would  use  the  above  signifiers,  and  in  particular  visual-graphical  ones 
during  dialogues  over  graphical  media  such  as  maps.  Since  military  planning  involves 
manual  references  to  significants  on  such  media  (McCann  &  Moogk,  1983),  we  should 
expect  the  use  of  visual-graphical  signifiers  in  denoting  the  figurative  properties  of  the 
significants. 


1.2  Objective 

This  study  aims  to  elucidate  the  figurative  relations  between  visual-graphical 
signifiers  and  the  significants  related  to  these  signifiers.  A  figurative  relation  would  imply 
that  the  visual-graphical  signifiers  have,  by  imitation,  a  physical  property  such  as  a 


3  A  mental  image  may  constitute  a  symbol  for  a  significant,  such  as  an  object  seen  in  the  past,  as 
well  as  act  as  the  signified  content,  i.e.,  a  significant  for  another  signifier. 


geometrical  form  that  reproduces  a  corresponding  property  of  their  significants.  The 
answer  to  this  issue  will  help  determine  the  symbolic  properties  of  visual-graphical 
signifiers,  and  their  use  in  communicating  spatial  significants  during  tactical  planning. 
Both  sources  of  information  will  provide  a  basis  upon  which  to  design  graphical 
interfaces  for  human-computer  multimodal  dialogues. 

2.  Method 

2.1  Subjects 

Seven  commanding  officers  participated  in  this  study  (McCann  &  Moogk,  1983). 
Some  of  the  participants  had  experience  in  commanding  actual  combat  operations  (e.g., 
in  the  second  world  war)  while  others  had  extensive  knowledge  of  Canadian  military 
doctrine.  Thus  the  participants  tapped  two  sources  of  spatial  information  relevant  for 
planning:  one  based  on  tactical  experience,  and  the  other  on  tactical  doctrine. 

2.2  Material 

A  topographic  map  of  1:50,000  scale  (Germany  series  M745)  depicted  the 
geographical  and  tactical  aspects  of  the  scenario  for  the  planning  problem.  A  plexiglas 
sheet  covering  the  map  allowed  subjects  to  mark  and  erase  graphical  annotations.  A  video 
camera,  mounted  to  the  ceiling  directly  above  the  topographic  map,  recorded  the  subjects' 
graphical  interactions  with  the  map  and  their  verbal  commentaries. 

2.3  Experimental  Procedure 

The  subjects  task  was  to  develop  and  orally  present  an  estimate  of  the  tactical 
scenario  depicted  on  the  spatial  display  from  the  perspective  of  the  Brigade  commander. 
Subjects  were  to  base  their  estimate  on  their  tactical  knowledge,  background  information4 
reviewed  prior  to  arriving  for  the  study,  and  the  Division  orders  and  intelligence  report 
that  were  now  available.  Subjects  were  permitted  to  mark  the  spatial  display  in  any  way 


4  This  package  consisted  of  a  sketch  map  of  the  tactical  area  (scale  1:250,000),  and  a  general 
concept  of  the  tactical  problem;  an  Intelligence  report  concerning  the  organization  and  equipment  of  own 
and  enemy  forces;  and  the  military  doctrine  of  enemy  forces.  Although  the  participants  were  familiar  with 
this  doctrine,  this  procedure  gave  them  a  common  basis  upon  which  to  perform  the  estimate  of  the  tactical 
situation. 


using  grease  pencils  provided.  The  tactical  scenario  and  method  are  described  in  more 
detail  in  McCann  &  Moogk  (1983). 

To  facilitate  the  participants'  presentations,  the  military  officer  who  had  written  the 
scenario  took  the  role  of  the  Regiment  Artillery  Commander,  who  is  often  present  during 
the  development  of  the  estimate.  (The  Regiment  is  tasked  by  the  Division  to  provide 
direct  artillery  support  to  the  Brigade.)  Upon  completion  of  the  oral  estimate,  the 
experimenters  reviewed  the  recordings  with  each  commander  to  clarify  the  nature  of  the 
spatial  information  (e.g.,  spatial  object,  concepts)  used.  The  length  of  the  sessions  varied 
between  1  hour  and  2  1/2  hours. 

3.  Results 

The  analysis  of  the  figurative  relations  between  the  visual-graphical  signifiers  and 
the  spatial  significants  is  based  on  two  sets  of  results:  (a)  the  lexicon  of  visual-graphical 
signifiers  (Boudreau  &  McCann,  1994),  and  (b)  the  total  set  of  spatial  significants  related 
to  these  signifiers  as  they  co-occurred  with  speech  (Boudreau  &  McCann,  1996).  These 
results  are  summarized  in  section  3.1.  Section  3.2  describes  the  figurative  relations 
between  the  signifiers  and  the  significants.  The  results  were  obtained  from  naturally- 
occurring  multimodal  dialogues,  but  for  this  reason  and  because  they  were  drawn  from  a 
small  sample  of  subjects,  they  must  be  considered  tentative. 

3.1  Visual-Graphical  Signifiers  and  Spatial  Significants 
3.1.1  Lexicon  of  Visual-Graphical  Signifiers 

Visual-graphical  signifiers  consisted  of  any  graphical  (hand)  action  involving  either 
a  manual  reference  to  a  significant  (e.g.,  geographical  object)  depicted  on  the  map;  or  the 
generation  of  a  significant,  whether  or  not  the  action  left  any  symbolic  mark  on  the 
display.  We  identified  a  lexicon  of  such  signifiers  from  the  total  set  of  graphical  actions 
that  commanders  used  in  the  tactical  planning  task.  We  developed  the  lexicon  by 
categorizing  the  graphical  actions  on  the  basis  of  the  physical  (geometrical  and  temporal) 
properties  that  visually  distinguished  the  actions.  The  categorization  approach  differed 
from  that  of  previous  studies  (Baraket,  1969;  Efron,  1941;  Ekman  &  Friesen,  1972; 
McNeill,  1985)  in  that  these  have  not  involved  graphical  media,  and  have  been  based  on 
assumptions  concerning  the  symbolic  properties  of  the  hand  motions.  For  example, 
certain  hand  motions  have  been  classified  as  iconic,  assuming  visual  similarity  between 
the  hand  motion  and  the  object  signified  (McNeill,  1985).  Our  classification  was  based 
purely  on  the  visual  observation  of  the  physical  properties  of  the  graphical  actions. 
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Geometrical  form  was  the  primary  visual  property  that  we  used  to  categorize  the 
actions.  We  distinguished  small  formless  "pointing"  motions  (either  with  finger  or  pen) 
from  motions  which  traced  a  one-dimensional  or  a  two-dimensional  form.  One¬ 
dimensional  actions  consisted  either  in  tracing  a  linear  form  (categorized  as  "linear")  or  a 
curvilinear  form  (categorized  as  "curvilinear").  Two-dimensional  actions  consisted  in 
covering  an  area  with  the  hand  or  length  of  pen  (categorized  as  "two-dimensional").  If  the 
action  resulted  in  a  visible  marking  of  the  map,  it  was  separately  categorized  as  a 
"drawing".  Thus  all  actions  were  classified  into  five  main  categories:  pointing,  one¬ 
dimensional  actions  that  are  either  curvilinear  (closed)  or  linear  (open),  two-dimensional 
(area-covering)  actions,  and  drawings. 

We  further  differentiated  one-dimensional  and  two-dimensional  actions  according 
to  their  momentum:  continuous  or  paused.  We  classified  an  action  as  continuous  when  it 
involved  an  uninterrupted  change  of  hand  position  but  a  constant  shape  during  the  apex 
of  the  motion.  During  this  temporal  phase,  the  motion  showed  a  distinct  peaking  of  effort 
observable  from  its  constant  shape  (Kendon,  1980).  In  contrast,  paused  motions  involved 
holding  the  hand  briefly  in  position  at  the  apex.  Drawings  were  intrinsically  continuous. 

Finally,  with  the  exception  of  drawings,  we  distinguished  the  actions  according  to 
their  bidirectional  properties.  We  considered  a  graphical  action  as  bidirectional  when  it 
involved  an  alternating  displacement  of  the  hand  (or  fingertip)  from  one  position  to 
another.  While  alternating  positions,  the  form  of  the  motion  remained  constant.  A 
bidirectional  pointing  motion  consisted  of  an  alternative  pointing  motion  directed  at  two 
different  objects  in  a  pair. 

The  lexicon  comprises  altogether  14  types  of  visual-graphical  signifiers  grouped 
within  the  five  categories  identified  above.  It  consists  of  pointing  actions  (single, 
repeated,  bidirectional);  one-dimensional  actions  that  are  either  curvilinear  (paused, 
continuous,  bidirectional)  or  linear  (paused,  continuous,  bidirectional);  two-dimensional 
area  covering  actions  (paused,  continuous,  bidirectional);  and  drawings  (symbols,  linear). 

3.1.2  Spatial  Significants 

We  determined  a  set  of  significants  from  the  analysis  of  the  commanders'  dialogues 
in  their  verbal  and  graphical  components.  These  dialogues  were  analysed  using  lexical, 
syntactical,  pragmatic  (planning  phases),  and  semantic  aspects  of  linguistic  analysis.  The 
significants  were  categorised,  in  a  representational  scheme,  along  two  dimensions:  spatial 
classes  and  type  (Boudreau  &  McCann,  1996). 


To  elucidate  the  figurative  properties  of  the  visual-graphical  signifiers,  we  first 
classified  the  significants  according  to  their  geometrical  attributes  (one-dimensional,  two- 
dimensional,  or  nonspecified  (N)  dimensionality),  and  then  according  to  the  categories 
and  types  of  signifiers  to  which  they  were  related. 

One-dimensional  significants  included: 

•  geographical  lines  (e.g.,  roads,  water  courses,  crest  lines,  ridges,  defiles), 

•  tactical  lines  (e.g.,  boundaries,  lines  of  operation,  avenues  of  approach),  and 

•  courses  of  actions  taken  along  these  lines  (e.g.,  tactical  operations  along  a  river, 
disposition  or  deployment  of  units  along  a  boundary). 

Two-dimensional  significants  included: 

•  geographical  areas  (e.g.,  area  of  vegetation,  high  grounds,  wooded  areas), 

•  tactical  areas  (e.g.,  key  terrains,  killing  ground(s),  areas  of  observation  and  fire, 
areas  of  concealment  and  cover),  and 

•  courses  of  actions  taken  within  these  areas  (e.g.,  tactical  operations  within  key 
terrains,  disposition  or  deployment  of  companies  along  areas  of  observation). 

N-dimensional  significants  comprised: 

•  combat  units  and  artillery,  and 

•  courses  of  actions  planned  for  these  combat  units  (e.g.,  tactical  operations  of  tank 
platoons,  disposition  or  deployment  of  company  positions). 

N-dimensional  significants  differed  from  the  other  two  classes  in  that  their 
dimensionally  was  specified  neither  verbally  nor  graphically  on  the  map  possibly  because 
of  the  military  symbology  denoting  these  significants,  or  their  geometrical  complexity. 
Table  1  relates  the  significants,  so  classified,  to  the  different  categories  and  types  of 
visual-graphical  signifiers.  The  percentages  are  calculated  from  the  total  number  of 
significants  related  to  a  type  of  signifier.  The  table  exhausts  the  whole  set  of  signifiers 
and  significants. 

As  shown,  pointing  actions  generally  designated  or  located  N-dimensional 
significants  (70%).  Linear  actions  and  linear  drawings  mainly  signified  one-dimensional 
significants  (64%  and  87%  respectively).  In  contrast,  curvilinear  and  two-dimensional 
actions  rarely  designated  one-dimensional  significants  (2%  and  5%  respectively).  Two- 
dimensional  actions  depicted  two-dimensional  significants  (60%)  more  frequently  than 


-7- 


Percentages  (%)  and  frequencies  (Fr)  of  significants  of  different  dimensionality  associated  with  each  category  and  type 
of  visual-graphical  signifier.  Numbers  in  the  column  headed  "N"  represent  significants  having  nonspecific 
dimensionality.  Numbers  in  each  row  should  sum  to  100. 


Visual-graphical  signifier 


Dimensionality  of  Significant 


Category 

Type 

One 

Two 

N 

Total  Fr 

% 

Fr 

% 

Fr 

% 

Fr 

Pointing 

Single 

13 

14 

17 

18 

70 

76 

108 

Repeated  (tapping) 

4 

1 

32 

8 

64 

16 

25 

Bidirectional  (two-point) 

0 

0 

20 

2 

80 

8 

10 

Combined  pointing 

10 

15 

20 

28 

70 

100 

143 

Curvilinear  (closed) 

Paused 

0 

0 

45 

5 

55 

6 

11 

Continuous 

0 

0 

43 

21 

28 

28 

49 

Bidirectional 

22 

2 

67 

6 

11 

1 

9 

Combined  closed 

2 

2 

48 

32 

50 

35 

69 

Linear (open) 

Paused 

57 

13 

13 

3 

30 

7 

23 

Continuous 

77 

55 

9 

6 

14 

10 

71 

Bidirectional 

36 

10 

36 

10 

28 

8 

28 

Combined  open 

64 

78 

15 

19 

21 

25 

122 

Two-dimensional 

Paused 

0 

0 

55 

11 

45 

9 

20 

(area  covering) 

Continuous 

9 

3 

60 

19 

31 

10 

32 

Bidirectional 

5 

1 

67 

12 

28 

5 

18 

Combined  area  covering 

5 

4 

60 

42 

35 

24 

70 

Drawings 

Symbols 

0 

0 

33 

3 

67 

6 

9 

Linear 

87 

34 

7 

3 

6 

2 

39 

Combined  drawings 

71 

34 

12 

6 

17 

8 

48 

N-dimensional  ones  (35%),  while  curvilinear  actions  illustrated  both  types  of  significants 
equally  (48%  and  50%  respectively).  These  results  indicate  that  visual-graphical  actions 
have  geometrical  attributes  that  depict  those  of  their  corresponding  significant(s). 

It  is  interesting  to  note  the  similarity  that  exists  between  the  geometrical  attributes 
of  the  visual-graphical  signifiers,  those  of  graphics  (Lee,  1991),  and  those  of  descriptors 
in  sign  languages  (Cuxac,  1991).  In  all  three  cases,  the  hand  action  specifies  geometrical 
attributes  of  significants  such  as  size  or  shape.  For  instance,  Lee  observed  that  the  shape 
of  hand  actions,  called  pointing,  changed  with  the  shape  of  the  significant(s)  referred  to: 
two-dimensional  motions  over  extended  areas,  or  linear  tracing  over  linear  boundaries. 
The  similarity  in  use  of  hand  actions,  in  different  contexts,  may  be  related  to  the  fact  that 
geometrical  concepts  and  mental  imagery  organize  and/or  influence  the  geometrical 
attributes  of  the  hand  actions. 


4.  Discussion 

This  study  aimed  at  elucidating  the  figurative  properties  of  the  visual-graphical 
signifiers  that  commanders  use  to  denote  significants  on  a  map  during  tactical  planning. 
The  results  are  twofold.  First,  they  indicate  that  pointing  signifiers  arbitrarily  relate  to 
their  significants  figurative  properties.  These  signifiers  act  rather  as  deictic  references 
which  co-occur  with  verbal  deictic  references,  such  as  this  or  there,  to  identify  and  locate 
significants  on  graphical  media  (Boudreau  &  McCann,  1994;  Kendon,  1985;  Levelt, 
Richardson,  &  Heij,  1985). 

In  contrast,  visual-graphical  signifiers  which  reproduce,  at  least  approximately,  the 
geometrical  attributes  of  their  significant(s)  have  figurative  properties.  These  signifiers 
include  one-dimensional  actions  that  are  either  curvilinear  or  linear,  two-dimensional 
actions,  and  drawings.  A  plausible  explanation  of  these  results  is  that  the  figurative 
properties  of  the  significants  partly  determine  those  of  their  related  signifiers  (Kendon, 
1985).  For  example,  if  the  referent  of  a  visual-graphical  signifier  is  linear,  the  motion 
imitates  the  linear  form  of  its  referent. 

One  can  further  argue  that,  in  real  time  cognitive  processing,  the  generation  of 
significants  precedes  and  partly  determines  the  production  of  signifiers  in  a  particular 
mode  (acoustic,  visual,  haptic)  adapted  to  convey  their  meaning  or  figurative  properties. 
Such  would  be  the  case  in  tasks  involving  graphical  media,  such  as  in  the  present,  where 
humans  use  visual  signifiers  to  depict  significants  available  on  the  media.  They  also  use 
these  visual  signifiers  to  represent  together  with  speech:  (1)  the  figurative  properties  of 
significants  that  are  already  known  but  not  actually  perceived  at  the  time  they  are 
symbolized;  and  (2)  those  of  novel  significants  that  have  been  deduced  or  imagined 
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during  planning.  For  example,  a  drawing  may  symbolize  the  mental  image  of  a  unit's 
disposition  seen  in  the  past,  but  it  may  also  represent  an  original  disposition  of  battle 
units.  Because  tactical  planning  involves  the  above  sorts  of  significants,  then  one  could 
suggest  that  visual-graphical  signifiers  are  actively  involved  with  speech,  to  communicate 
significants  during  tactical  planning.  Visual-graphical  signifiers  would  thus  symbolise  the 
concrete  or  visual  properties  of  significants  in  a  way  similar  to  which  verbal  signifiers 
symbolise  abstract  properties  of  significants.  The  complementary  role  of  visual-graphical 
and  verbal  signifiers  reflects  the  coordinated  and  dual  nature  of  the  mental  representation 
that  occurs  during  dialogues  involving  graphical  media  (Jeannerod,  1994;  Paivio,  1986; 
Piaget,  1983;  Piaget  &  Inhelder,  1963). 


5.  Conclusions 

The  results  of  this  study  suggest  that  visual-graphical  signifiers  provide,  when 
deferred,  symbols  for  visual  significants,  extending  some  of  these,  such  as  mental  images, 
into  observable  behaviors  that  are  not  easily  conveyed  through  speech.  The  figurative 
properties  of  visual-graphical  signifiers  thus  support  the  utility  of  designing  graphical 
interfaces  to  recognise  these  actions.  Rhyne  and  Wolf  (1986)  propose  two  approaches  to 
the  design  of  graphical  interfaces:  one  is  user-independent,  and  the  other  user-dependent. 
In  the  first  approach,  one  would  analyse  the  figurative  properties  of  hand  motions  from  a 
large  number  of  subjects,  and  then  create  templates  against  which  to  compare  an 
individual's  production.  The  efficiency  of  this  approach  depends  on  inter-subject 
consistency,  i.e.,  the  extent  to  which  there  is  consistency  between  subjects  in  the  use  of 
the  figurative  properties  of  the  hand  motions.  In  the  second  approach,  the  user  would  train 
the  interface  in  advance  on  a  set  of  hand  motions  that  he  (she)  will  use.  For  accurate 
recognition,  each  user  must  reproduce  the  trained  set  of  motions  consistently  in  the  same 
geometrical  form.  The  efficiency  of  the  user-dependent  approach  thus  depends  on  intra¬ 
subject  consistency. 

The  subjects  in  this  study  generated  a  common  lexicon  of  signifiers,  plus  individual 
additions  or  variations  (Boudreau  &  McCann,  1994).  They  also  produced,  via  imitation  or 
deferred  imitation,  a  consistent  set  of  figurative  properties  to  denote  significants.  Given 
these  symbolic  properties,  one  could  design  a  graphical  interface  that  combines  the  above 
two  approaches.  With  the  user-independent  approach,  a  first  step,  prior  to  the  design  of  a 
graphical  interface,  would  be  to  verify  inter-subject  consistency  in  the  use  of  the  lexicon 
from  a  large  sample  of  subjects.  This  information  would  then  form  a  basis  to  create 
standardized  models  of  the  signifiers  in  terms  of  their  geometrical  form  and  meaning. 
With  the  user-dependent  approach,  the  interface  would  adapt  to  individual  differences  in 


production  of  the  signifiers.  Wolf,  Rhyne  and  Ellozy  (1989)  adopted  the  above  two 
approaches  to  design  a  Paper-Like  interface  that  can  recognize  written  symbols  as  well  as 
a  user's  own  style  of  writing  for  any  symbol  set.  Similarly,  Marukami  and  Taguchi  (1991) 
trained  neural  networks  to  recognise  an  individual's  way  of  signing  in  Japanese  using 
datagloves. 

Paper-Like  interfaces  recognise  two-dimensional  (2D)  motions,  such  as  hand-drawn 
symbols,  while  datagloves  are  tailored  for  three-dimensional  (3D)  motions.  Since  visual- 
graphical  signifiers  have  2D  and  3D  components,  graphical  interfaces  should  integrate  the 
two  technologies.  Such  graphical  interfaces  would  allow  the  user  to  employ  any  part  of 
his  hand(s)  to  denote  spatial  significants  on  a  2D  surface  as  observed  in  this  study,  but 
also  to  generate  significants  in  a  3D  space.  The  design  of  such  a  graphical  interface  would 
constitute  an  integral  part  of  a  multimodal  interface  capable  of  recognising  and 
interpreting  both  verbal  and  visual-graphical  signifiers  concurrently. 
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